研究背景及论文工作概述
场景图是一种图结构，节点代表图像中的实体（物体），边表示实体之间的关系。场景图生成（Scene Graph Generation, SGG）是根据一张给定的图像，生成对应的场景图，这要求模型不仅要识别出图中的实体，还要检测实体之间的语义关系。场景图生成超越了普通的目标检测任务，与视觉关系检测紧密相关，并在多模态任务中展现出潜力。
当前主流的场景图生成方法基于两阶段流程。模型首先使用检测器生成实体候选框，再通过神经网络对物体候选框之间的关系进行分类。这种方式能够实现良好的效果，但面临着模型参数量大、计算复杂度高等（O(n^2)）问题。
RelTR提出了一种单阶段的端到端场景图生成方法。在架构上，RelTR基于 encoder-decoder架构。编码器负责提取图像的视觉特征表示，解码器基于预训练的查询特征和多种注意力机制直接预测一组固定数量的关系三元组。
论文创新点
对比以往方法，RelTR的主要贡献和创新点如下：
- 对比以往的两阶段场景图生成方法，RelTR 能够通过视觉信息直接生成稀疏的场景图，具备更小的参数和计算复杂度。如下图，RelTR 只需要预测固定数量的关系三元组，不需要预测所有实体对之间的关系。
- 论文设计了一种集合预测损失（set prediction loss），通过基于IoU的分配策略，将预测的三元组与真实标注进行匹配，让模型直接预测三元组成为可能。下面的公式是 RelTR 的损失表达式，后文会进行更细致的介绍。
- RelTR的三元组解码器能够利用实体解码器输出的实体检测结果，进一步提升主客体的定位与分类精度。
RelTR
在这一部分，本文将对RelTR的实现进行更加详细的介绍。
下图是 RelTR 的整体架构。RelTR 基于 encoder-decoder 的架构。RelTR 包括三个核心组件：用于提取视觉特征上下文信息的特征编码器、基于DETR框架捕获实体表征的实体解码器以及三元组解码器。其中RelTR的主要创新在于三元组解码器，接下来我们对其三元组解码器进行介绍。
三元组解码器
Subject and Object Queries
RelTR 引入了固定数目的主语查询和宾语查询。这些查询是学习后的 Embedding。主语查询和宾语查询之间一一对应。RelTR 通过 <主语-宾语-关系>来表示稀疏的场景图。此外，在 RelTR 学习过程中，也引入了可训练的 E_o 和 E_t。
多种注意力机制
RelTR 设计了** 耦合自注意力机制（CSA）**、**解耦视觉注意力（DVA）和 解耦实体注意力（DEA）。**三种注意力计算方式分别如下：
- CSA：
- DVA：
- DEA：
其中 Q_s 是主语查询特征表示，Q_o是宾语查询特征表示。E_o 和 E_t 是可学习的主语/宾语编码。
set prediction loss
RelTR扩展了DETR的损失计算方式，设计了专门用于三元组检测的集合预测损失。该损失函数通过计算三元组预测（包含主/客体置信度、谓词置信度及边界框IoU）的cost矩阵，将真实三元组分配给最匹配的预测。未被匹配的预测会被赋予<背景-无关系-背景>标签。
针对训练中出现的多个查询聚焦同一真实三元组的问题（如预测A和D同时匹配同一目标，但D因稍高cost被误判为背景），RelTR提出了基于IoU的分配策略改进：当预测的主/客体类别正确且边界框IoU超过阈值时，即使未被匈牙利算法匹配，也不计算该主/客体的预测损失。这一策略有效避免了因局部预测误差（如关系分类错误）导致优质检测结果被错误惩罚的问题。
基于MindSpore评估模型
实验环境
NPU：1 * Ascend 910（显存：32）,CPU: 24, 内存：96GB
mindspore 2.50 + python 3.9 + Cann 7.0RC1
基于MindSpore实现模型推理
基于MindSpore实现RelTR模型（原代码使用了自定义的Transformer和改进的ResNet，全部展示过长，基于MindSpore实现的代码已发布到gitee仓库：https://gitee.com/yanrui2025/ms-rel-tr）：
class RelTR(nn.Cell):
""" RelTR: Relation Transformer for Scene Graph Generation """
def __init__(self, backbone, transformer, num_classes, num_rel_classes, num_entities, num_triplets, aux_loss=False, matcher=None):
""" Initializes the model.
Parameters:
backbone: torch module of the backbone to be used. See backbone.py
transformer: torch module of the transformer architecture. See transformer.py
num_classes: number of entity classes
num_entities: number of entity queries
num_triplets: number of coupled subject/object queries
aux_loss: True if auxiliary decoding losses (loss at each decoder layer) are to be used.
"""
super().__init__()
self.num_entities = num_entities
self.transformer = transformer
hidden_dim = transformer.d_model
self.hidden_dim = hidden_dim
self.input_proj = nn.Conv2d(backbone.num_channels, hidden_dim, kernel_size=1, has_bias=True)
self.backbone = backbone
self.aux_loss = aux_loss
self.entity_embed = nn.Embedding(num_entities, hidden_dim*2)
self.triplet_embed = nn.Embedding(num_triplets, hidden_dim*3)
self.so_embed = nn.Embedding(2, hidden_dim) # subject and object encoding
# entity prediction
self.entity_class_embed = nn.Dense(hidden_dim, num_classes + 1)
self.entity_bbox_embed = MLP(hidden_dim, hidden_dim, 4, 3)
# mask head
self.so_mask_conv = nn.SequentialCell([nn.Upsample(size=(28, 28)),
nn.Conv2d(2, 64, kernel_size=3, stride=2, pad_mode='pad', padding=3, has_bias=True),
nn.ReLU(),
nn.BatchNorm2d(64),
nn.MaxPool2d(kernel_size=3, pad_mode='pad', stride=2, padding=1),
nn.Conv2d(64, 32, kernel_size=3, stride=1, pad_mode='pad', padding=1, has_bias=True),
nn.ReLU(),
nn.BatchNorm2d(32)])
self.so_mask_fc = nn.SequentialCell([nn.Dense(2048, 512),
nn.ReLU(),
nn.Dense(512, 128)])
# predicate classification
self.rel_class_embed = MLP(hidden_dim*2+128, hidden_dim, num_rel_classes + 1, 2)
# subject/object label classfication and box regression
self.sub_class_embed = nn.Dense(hidden_dim, num_classes + 1)
self.sub_bbox_embed = MLP(hidden_dim, hidden_dim, 4, 3)
self.obj_class_embed = nn.Dense(hidden_dim, num_classes + 1)
self.obj_bbox_embed = MLP(hidden_dim, hidden_dim, 4, 3)
def construct(self, samples):
""" The forward expects a NestedTensor, which consists of:
- samples.tensor: batched images, of shape [batch_size x 3 x H x W]
- samples.mask: a binary mask of shape [batch_size x H x W], containing 1 on padded pixels
It returns a dict with the following elements:
- "pred_logits": the entity classification logits (including no-object) for all entity queries.
Shape= [batch_size x num_queries x (num_classes + 1)]
- "pred_boxes": the normalized entity boxes coordinates for all entity queries, represented as
(center_x, center_y, height, width). These values are normalized in [0, 1],
relative to the size of each individual image (disregarding possible padding).
See PostProcess for information on how to retrieve the unnormalized bounding box.
- "sub_logits": the subject classification logits
- "obj_logits": the object classification logits
- "sub_boxes": the normalized subject boxes coordinates
- "obj_boxes": the normalized object boxes coordinates
- "aux_outputs": Optional, only returned when auxilary losses are activated. It is a list of
dictionnaries containing the two above keys for each decoder layer.
"""
if isinstance(samples, (list, Tensor)):
samples = nested_tensor_from_tensor_list(samples)
features, pos = self.backbone(samples)
src, mask = features[-1].decompose()
assert mask is not None
hs, hs_t, so_masks, _ = self.transformer(self.input_proj(src), mask, self.entity_embed.embedding_table,
self.triplet_embed.embedding_table, pos[-1], self.so_embed.embedding_table) # bug input_proj
so_masks = self.so_mask_conv(so_masks.view(-1, 2, src.shape[-2],src.shape[-1])).view(hs_t.shape[0], hs_t.shape[1], hs_t.shape[2],-1)
so_masks = self.so_mask_fc(so_masks)
split = ops.Split(axis=-1, output_num=2)
hs_sub, hs_obj = split(hs_t)
outputs_class = self.entity_class_embed(hs)
outputs_coord = self.entity_bbox_embed(hs).sigmoid()
outputs_class_sub = self.sub_class_embed(hs_sub)
outputs_coord_sub = self.sub_bbox_embed(hs_sub).sigmoid()
outputs_class_obj = self.obj_class_embed(hs_obj)
outputs_coord_obj = self.obj_bbox_embed(hs_obj).sigmoid()
concat = ops.Concat(axis=-1)
concat_feat = concat((hs_sub, hs_obj, so_masks))
outputs_class_rel = self.rel_class_embed(concat_feat)
out = {'pred_logits': outputs_class[-1], 'pred_boxes': outputs_coord[-1],
'sub_logits': outputs_class_sub[-1], 'sub_boxes': outputs_coord_sub[-1],
'obj_logits': outputs_class_obj[-1], 'obj_boxes': outputs_coord_obj[-1],
'rel_logits': outputs_class_rel[-1]}
if self.aux_loss:
out['aux_outputs'] = self._set_aux_loss(outputs_class, outputs_coord, outputs_class_sub, outputs_coord_sub,
outputs_class_obj, outputs_coord_obj, outputs_class_rel)
return out
# @torch.jit.unused
def _set_aux_loss(self, outputs_class, outputs_coord, outputs_class_sub, outputs_coord_sub,
outputs_class_obj, outputs_coord_obj, outputs_class_rel):
# this is a workaround to make torchscript happy, as torchscript
# doesn't support dictionary with non-homogeneous values, such
# as a dict having both a Tensor and a list.
return [{'pred_logits': a, 'pred_boxes': b, 'sub_logits': c, 'sub_boxes': d, 'obj_logits': e, 'obj_boxes': f,
'rel_logits': g}
for a, b, c, d, e, f, g in zip(outputs_class[:-1], outputs_coord[:-1], outputs_class_sub[:-1],
outputs_coord_sub[:-1], outputs_class_obj[:-1], outputs_coord_obj[:-1],
outputs_class_rel[:-1])]
基于MindSpore实现数据加载。这里实现了数据的加载以及预处理。
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
# Copyright (c) Institute of Information Processing, Leibniz University Hannover.
"""
dataset (COCO-like) which returns image_id for evaluation.
Mostly copy-paste from https://github.com/pytorch/vision/blob/13b35ff/references/detection/coco_utils.py
"""
from pathlib import Path
import json
#
import numpy as np
import mindspore.numpy as mnp
from pycocotools import mask as coco_mask
import os
import numpy as np
from PIL import Image
from pycocotools.coco import COCO
from mindspore import Tensor
import mindspore as ms
import mindspore.ops as ops
import mindspore.dataset.vision.c_transforms as c_vision
import mindspore.dataset.vision.py_transforms as py_vision
# from mindspore.dataset.transforms.py_transforms import Compose
from mindspore.dataset.transforms import Compose
import mindspore.dataset as ds
import cv2
class CocoDetection:
def __init__(self, img_folder, ann_file, transforms=None, return_masks=False):
self.img_folder = img_folder
self.coco = COCO(ann_file)
self.ids = list(sorted(self.coco.imgs.keys()))
self._transforms = transforms
self.prepare = ConvertCocoPolysToMask(return_masks)
# 加载关系标注
rel_json_path = '/'.join(ann_file.split('/')[:-1]) + '/rel.json'
with open(rel_json_path, 'r') as f:
all_rels = json.load(f)
if 'train' in ann_file:
self.rel_annotations = all_rels['train']
elif 'val' in ann_file:
self.rel_annotations = all_rels['val']
else:
self.rel_annotations = all_rels['test']
self.rel_categories = all_rels['rel_categories']
def __getitem__(self, idx):
"""
返回图像和标注（包括关系标注）
"""
print(idx)
image_id = self.ids[idx]
ann_ids = self.coco.getAnnIds(imgIds=image_id)
target = self.coco.loadAnns(ann_ids) # 加载标注
# 加载图像
img_info = self.coco.loadImgs(image_id)[0]
img_path = os.path.join(self.img_folder, img_info['file_name'])
img = Image.open(img_path).convert('RGB')
# 加载关系标注
rel_target = self.rel_annotations[str(image_id)]
# 构造target字典（和PyTorch版本一致）
target_dict = {
'image_id': image_id,
'annotations': target,
'rel_annotations': rel_target
}
# 应用标注转换（如将多边形转换为mask）
img, target_dict = self.prepare(img, target_dict)
# 应用图像变换（如Resize、ToTensor等）
if self._transforms is not None:
img, target_dict = self._transforms(img, target_dict)
return img, target_dict
def __len__(self):
return len(self.ids)
def convert_coco_poly_to_mask(segmentations, height, width):
masks = []
for polygons in segmentations:
# 1. 将多边形转换为RLE格式
rles = coco_mask.frPyObjects(polygons, height, width)
# 2. 解码RLE为mask（numpy数组）
mask = coco_mask.decode(rles) # shape: [H, W] 或 [num_instances, H, W]
# 3. 处理mask维度（如果只有1个实例，增加一个维度）
if len(mask.shape) < 3:
mask = mask[..., None] # shape: [H, W, 1]
# 4. 转换为MindSpore的Tensor（注意：这里先用numpy处理，再转MindSpore）
mask = mnp.tensor(mask, dtype=mnp.uint8)
# 5. 沿通道维度取任意值（mask.any(dim=2)）
mask = mnp.any(mask, axis=2) # shape: [H, W]
masks.append(mask)
# 6. 堆叠所有mask（如果masks非空）
if masks:
masks = mnp.stack(masks, axis=0) # shape: [N, H, W]
else:
masks = mnp.zeros((0, height, width), dtype=mnp.uint8)
return masks
class ConvertCocoPolysToMask:
def __init__(self, return_masks=False):
self.return_masks = return_masks
def __call__(self, image, target):
w, h = image.size
image_id = target["image_id"]
image_id = Tensor([image_id], dtype=ms.int64) # MindSpore Tensor
anno = target["annotations"]
anno = [obj for obj in anno if 'iscrowd' not in obj or obj['iscrowd'] == 0]
boxes = [obj["bbox"] for obj in anno]
if not boxes: # 处理空边界框的情况
boxes = Tensor([], dtype=ms.float32).reshape(0, 4)
else:
boxes = Tensor(boxes, dtype=ms.float32).reshape(-1, 4)
# 转换格式并裁剪坐标
boxes[:, 2:] += boxes[:, :2] # (x, y, w, h) -> (x1, y1, x2, y2)
boxes[:, 0::2] = boxes[:, 0::2].clip(0, w) # 替换 torch.clamp_
boxes[:, 1::2] = boxes[:, 1::2].clip(0, h)
classes = [obj["category_id"] for obj in anno]
classes = Tensor(classes, dtype=ms.int64)
masks, keypoints = None, None
if self.return_masks:
segmentations = [obj["segmentation"] for obj in anno]
masks = convert_coco_poly_to_mask(segmentations, h, w) # 假设已实现 MindSpore 版本
if anno and "keypoints" in anno[0]:
keypoints = [obj["keypoints"] for obj in anno]
keypoints = Tensor(keypoints, dtype=ms.float32)
num_keypoints = keypoints.shape[0]
if num_keypoints:
keypoints = keypoints.reshape(num_keypoints, -1, 3)
# 创建保留索引的条件
keep = ops.logical_and(boxes[:, 3] > boxes[:, 1],
boxes[:, 2] > boxes[:, 0])
if keep.size > 0: # 确保有元素可索引
boxes = boxes[keep]
classes = classes[keep]
if masks is not None:
masks = masks[keep]
if keypoints is not None:
keypoints = keypoints[keep]
# 构建新的 target 字典
rel_annotations = target['rel_annotations']
new_target = {
"boxes": boxes,
"labels": classes,
"image_id": image_id,
"keypoints": keypoints if keypoints is not None else Tensor([]),
"rel_annotations": Tensor(rel_annotations, dtype=ms.int32)
}
if self.return_masks and masks is not None:
new_target["masks"] = masks
# 添加额外字段
area = Tensor([obj["area"] for obj in anno], dtype=ms.float32)
iscrowd = Tensor([obj.get("iscrowd", 0) for obj in anno], dtype=ms.int64)
if keep.size > 0:
new_target["area"] = area[keep]
new_target["iscrowd"] = iscrowd[keep]
else:
new_target["area"] = Tensor([], dtype=ms.float32)
new_target["iscrowd"] = Tensor([], dtype=ms.int64)
new_target["orig_size"] = Tensor([h, w], dtype=ms.int64)
new_target["size"] = Tensor([h, w], dtype=ms.int64)
return image, new_target
#----------------------------------------------------------------#
class RandomHorizontalFlip:
def __init__(self, prob=0.5):
self.prob = prob
self.flip_op = c_vision.RandomHorizontalFlip(prob=prob)
def __call__(self, image, target):
if np.random.rand() < self.prob:
image = self.flip_op(image)
if "boxes" in target:
boxes = target["boxes"]
# Flip boxes coordinates (x1, y1, x2, y2)
w = image.shape[1]
boxes[:, [0, 2]] = w - boxes[:, [2, 0]]
target["boxes"] = boxes
return image, target
class RandomSelect:
def __init__(self, transform1, transform2, prob=0.5):
self.transform1 = transform1
self.transform2 = transform2
self.prob = prob
def __call__(self, image, target):
if np.random.rand() < self.prob:
return self.transform1(image, target)
else:
return self.transform2(image, target)
class RandomResize:
def __init__(self, scales, max_size=1333):
self.sizes = scales
self.max_size = max_size
def __call__(self, image, target):
size = np.random.choice(self.sizes)
image = np.array(image)
# 保持宽高比调整大小
h, w, _ = image.shape
scale = size / min(h, w)
if max(h, w) * scale > self.max_size:
scale = self.max_size / max(h, w)
new_h, new_w = int(h * scale), int(w * scale)
image = cv2.resize(image, (new_w, new_h))
# 调整边界框坐标
if "boxes" in target:
boxes = target["boxes"]
boxes[:, [0, 2]] *= new_w / w
boxes[:, [1, 3]] *= new_h / h
target["boxes"] = boxes
# 确保边界框在图像范围内
boxes[:, 0::2] = ops.clip_by_value(boxes[:, 0::2], ops.Tensor(0), ops.Tensor(new_w))
boxes[:, 1::2] = ops.clip_by_value(boxes[:, 1::2], ops.Tensor(0), ops.Tensor(new_h))
target["boxes"] = boxes
# 更新图像大小信息
if "size" in target:
target["size"] = Tensor([new_h, new_w], dtype=ms.float32)
return image, target
class ToTensor:
def __call__(self, image, target):
# 将图像从HWC转为CHW并归一化到[0,1]
image = image.transpose((2, 0, 1)) / 255.0
image = ms.Tensor(image, dtype=ms.float32)
return image, target
import mindspore as ms
from mindspore import Tensor, ops
import numpy as np
class Normalize:
def __init__(self, mean, std):
"""
Args:
mean (list/tuple): 各通道的均值 (R, G, B)
std (list/tuple): 各通道的标准差 (R, G, B)
"""
# 转换为MindSpore Tensor并重塑形状为(C,1,1)以便广播
self.mean = Tensor(mean, ms.float32).reshape(3, 1, 1)
self.std = Tensor(std, ms.float32).reshape(3, 1, 1)
# 获取MindSpore操作算子
self.sub = ops.Sub()
self.div = ops.Div()
self.cast = ops.Cast()
def __call__(self, image, target):
"""
Args:
image (Tensor): 输入图像张量 (C,H,W)
target: 目标数据（可以是任意类型）
Returns:
tuple: (归一化后的图像, 目标)
"""
# 确保输入是Tensor
if isinstance(image, np.ndarray):
image = Tensor(image, ms.float32)
# 执行归一化 (image - mean) / std
image = self.sub(image, self.mean)
image = self.div(image, self.std)
# 确保输出为float32类型
image = self.cast(image, ms.float32)
return image, target
def make_coco_transforms(image_set):
if image_set == 'train':
return Compose([
RandomHorizontalFlip(),
RandomSelect(
RandomResize(scales=[480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800], max_size=1333),
Compose([
RandomResize(scales=[400, 500, 600], max_size=1333),
# 注意：这里移除了裁剪操作，因为它可能会导致关系标注错误
RandomResize(scales=[480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800], max_size=1333)
])
),
ToTensor(),
Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])
elif image_set == 'val':
return Compose([
RandomResize(scales=[800], max_size=1333),
ToTensor(),
Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])
raise ValueError(f'unknown {image_set}')
#-------------------------end-------------------------------------#
def build():
ann_path = './data/vg/'
ann_path = ann_path
img_folder = '/home/ma-user/work/dataset/VG_100K/VG_100K/'
ann_file = ann_path + 'val.json'
dataset = CocoDetection(img_folder, ann_file, transforms=make_coco_transforms('val'), return_masks=False)
return dataset
为了实现模型的评估，需要将模型输出的原始张量转换为COCO评估格式：
class PostProcess(nn.Cell):
""" This module converts the model's output into the format expected by the coco api"""
def construct(self, outputs, target_sizes):
""" Perform the computation
Parameters:
outputs: raw outputs of the model
target_sizes: tensor of dimension [batch_size x 2] containing the size of each images of the batch
For evaluation, this must be the original image size (before any data augmentation)
For visualization, this should be the image size after data augment, but before padding
"""
out_logits, out_bbox = outputs['pred_logits'], outputs['pred_boxes']
assert len(out_logits) == len(target_sizes)
assert target_sizes.shape[1] == 2
softmax = ops.Softmax(axis=-1)
prob = softmax(out_logits)
scores, labels = prob[..., :-1].max(-1)
# convert to [x0, y0, x1, y1] format
boxes = box_ops.box_cxcywh_to_xyxy(out_bbox)
# and from relative [0, 1] to absolute [0, height] coordinates
img_h, img_w = target_sizes.unbind(1)
stack = ops.Stack(axis=1)
scale_fct = stack([img_w, img_h, img_w, img_h])
boxes = boxes * scale_fct[:, None, :]
results = [{'scores': s, 'labels': l, 'boxes': b} for s, l, b in zip(scores, labels, boxes)]
return results
进行数据推理及结果的评估：
def evaluate(model, criterion, postprocessors, data_loader, base_ds):
dataset = 'vg'
eval = True
model.set_train(False)
criterion.set_train(False)
metric_logger = utils.MetricLogger(delimiter=" ")
metric_logger.add_meter('class_error', utils.SmoothedValue(window_size=1, fmt='{value:.2f}'))
metric_logger.add_meter('sub_error', utils.SmoothedValue(window_size=1, fmt='{value:.2f}'))
metric_logger.add_meter('obj_error', utils.SmoothedValue(window_size=1, fmt='{value:.2f}'))
metric_logger.add_meter('rel_error', utils.SmoothedValue(window_size=1, fmt='{value:.2f}'))
header = 'Test:'
# initilize evaluator
# TODO merge evaluation programs
if dataset == 'vg':
evaluator = BasicSceneGraphEvaluator.all_modes(multiple_preds=False)
if eval:
evaluator_list = []
for index, name in enumerate(data_loader.dataset.rel_categories):
if index == 0:
continue
evaluator_list.append((index, name, BasicSceneGraphEvaluator.all_modes()))
else:
evaluator_list = None
else:
all_results = []
iou_types = tuple(k for k in ('segm', 'bbox') if k in postprocessors.keys())
coco_evaluator = CocoEvaluator(base_ds, iou_types)
for samples, targets in metric_logger.log_every(data_loader, 100, header):
# samples = samples.to(device)
# targets = [{k: v.to(device) for k, v in t.items()} for t in targets]
outputs = model(samples)
loss_dict = criterion(outputs, targets)
weight_dict = criterion.weight_dict
# reduce losses over all GPUs for logging purposes
loss_dict_reduced = loss_dict
loss_dict_reduced_scaled = {k: v * weight_dict[k]
for k, v in loss_dict_reduced.items() if k in weight_dict}
loss_dict_reduced_unscaled = {f'{k}_unscaled': v
for k, v in loss_dict_reduced.items()}
metric_logger.update(loss=sum(loss_dict_reduced_scaled.values()),
**loss_dict_reduced_scaled,
**loss_dict_reduced_unscaled)
metric_logger.update(class_error=loss_dict_reduced['class_error'])
metric_logger.update(sub_error=loss_dict_reduced['sub_error'])
metric_logger.update(obj_error=loss_dict_reduced['obj_error'])
metric_logger.update(rel_error=loss_dict_reduced['rel_error'])
if dataset == 'vg':
evaluate_rel_batch(outputs, targets, evaluator, evaluator_list)
orig_target_sizes = torch.stack([t["orig_size"] for t in targets], dim=0)
results = postprocessors['bbox'](outputs, orig_target_sizes)
res = {target['image_id'].item(): output for target, output in zip(targets, results)}
if coco_evaluator is not None:
coco_evaluator.update(res)
if dataset == 'vg':
evaluator['sgdet'].print_stats()
if eval and dataset == 'vg':
calculate_mR_from_evaluator_list(evaluator_list, 'sgdet')
# gather the stats from all processes
metric_logger.synchronize_between_processes()
print("Averaged stats:", metric_logger)
if coco_evaluator is not None:
coco_evaluator.synchronize_between_processes()
# accumulate predictions from all images
if coco_evaluator is not None:
coco_evaluator.accumulate()
coco_evaluator.summarize()
stats = {k: meter.global_avg for k, meter in metric_logger.meters.items()}
if coco_evaluator is not None:
if 'bbox' in postprocessors.keys():
stats['coco_eval_bbox'] = coco_evaluator.coco_eval['bbox'].stats.tolist()
return stats, coco_evaluator
def evaluate_rel_batch(outputs, targets, evaluator, evaluator_list):
for batch, target in enumerate(targets):
target_bboxes_scaled = rescale_bboxes(target['boxes'].cpu(), torch.flip(target['orig_size'],dims=[0]).cpu()).clone().numpy() # recovered boxes with original size
gt_entry = {'gt_classes': target['labels'].cpu().clone().numpy(),
'gt_relations': target['rel_annotations'].cpu().clone().numpy(),
'gt_boxes': target_bboxes_scaled}
sub_bboxes_scaled = rescale_bboxes(outputs['sub_boxes'][batch].cpu(), torch.flip(target['orig_size'],dims=[0]).cpu()).clone().numpy()
obj_bboxes_scaled = rescale_bboxes(outputs['obj_boxes'][batch].cpu(), torch.flip(target['orig_size'],dims=[0]).cpu()).clone().numpy()
pred_sub_scores, pred_sub_classes = torch.max(outputs['sub_logits'][batch].softmax(-1)[:, :-1], dim=1)
pred_obj_scores, pred_obj_classes = torch.max(outputs['obj_logits'][batch].softmax(-1)[:, :-1], dim=1)
rel_scores = outputs['rel_logits'][batch][:,1:-1].softmax(-1)
pred_entry = {'sub_boxes': sub_bboxes_scaled,
'sub_classes': pred_sub_classes.cpu().clone().numpy(),
'sub_scores': pred_sub_scores.cpu().clone().numpy(),
'obj_boxes': obj_bboxes_scaled,
'obj_classes': pred_obj_classes.cpu().clone().numpy(),
'obj_scores': pred_obj_scores.cpu().clone().numpy(),
'rel_scores': rel_scores.cpu().clone().numpy()}
evaluator['sgdet'].evaluate_scene_graph_entry(gt_entry, pred_entry)
if evaluator_list is not None:
for pred_id, _, evaluator_rel in evaluator_list:
gt_entry_rel = gt_entry.copy()
mask = np.in1d(gt_entry_rel['gt_relations'][:, -1], pred_id)
gt_entry_rel['gt_relations'] = gt_entry_rel['gt_relations'][mask, :]
if gt_entry_rel['gt_relations'].shape[0] == 0:
continue
evaluator_rel['sgdet'].evaluate_scene_graph_entry(gt_entry_rel, pred_entry)
测试结果及可视化展示
在 Visual Genome 数据集上进行测试。实验结果如下：
| R@20 | R@50 | mR@20 | mR@50 | zsR@50 | zsR@100 | Avg | |
|---|---|---|---|---|---|---|---|
| 基于pytorch | 21.2 | 27.5 | 6.8 | 10.8 | 1.8 | 2.4 | 11.8 |
| 基于mindspore | 21.0 | 27.4 | 6.6 | 10.8 | 1.8 | 2.3 | 11.7 |
下面是由mindspore模型输出的可视化结果：
一些小插曲
模型迁移问题：
在一开始测试时，我们选择直接将pytorch预训练好的模型加载到复现的MindSpore模型上。而在将迁移后的MindSpore模型参数保存为ckpt文件后后重新加载时，MindSpore模型的输出是异常的。这可能是ckpt转化的问题，用下面的方式直接根据参数名称映射赋值则没有问题。
mindspore_to_pytorch_mapping = pd.read_csv('mindspore_to_pytorch_mapping.csv')
mapping_dict = dict(zip(mindspore_to_pytorch_mapping['MindSpore Param'],
mindspore_to_pytorch_mapping['PyTorch Param']))
ms_values_dict = {}
for i in ms_param_names:
ms_key = i
pt_key = mapping_dict[ms_key]
pt_val = state['model'][pt_key]
if not isinstance(pt_val, np.ndarray):
pt_val = pt_val.cpu().numpy()
ms_val = Parameter(pt_val, ms_key)
ms_values_dict[ms_key] = ms_val
MindSpore模型计算问题：
在本地CPU上使用模型推理时，对于同一个图片，pytorch模型能够正常推理，而MindSpore模型会输出nan。
pytorch模型的输出：
MindSpore模型的输出：
两者前半部分的输出是相同的，说明模型的结构和参数并没有问题。问题可能出在MindSpore版本和运行的硬件环境上。在启智平台基于Ascend 910 使用MindSpore模型推理则无异常。
参考
PAMI | 基于Transformer的单阶段场景图生成：https://blog.csdn.net/python_plus/article/details/139517782
场景图生成网络——RelTR（TPAMI2023）：https://blog.csdn.net/python_plus/article/details/139517782